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Executive  Summary 


For  a  cybersecurity  workforce  to  be  effective,  its  members  must  possess  the  knowledge,  skills, 
and  experience  required  to  perform  their  job  duties.  Proficiency  and  relevance  are  key  factors  in 
determining  the  effectiveness  of  each  of  these  components.  Proficiency  refers  to  how  well 
someone  understands  a  subject  matter  or  can  apply  a  given  skill.  Relevance  refers  to  how  useful 
that  knowledge  or  skill  is  in  performing  a  given  job  duty.  For  example,  someone  could  have 
expert-level  knowledge,  skill,  and  experience  in  a  particular  area,  but  those  assets  will  have 
minimal  bearing  on  performance  if  the  person’s  area  of  expertise  is  not  relevant  to  their  job  duties. 

Organizations  are  faced  with  the  ongoing  challenge  of  ensuring  that  their  current  workforce 
possesses  the  most  current  knowledge,  skills,  and  experiences — with  an  emphasis  on  proficiency 
and  relevance.  However,  this  issue  is  particularly  challenging  for  a  cybersecurity  workforce 
because  industry  trends,  practices,  and  technologies  are  constantly  changing.  For  example,  cyber 
attack  vectors  are  constantly  changing  as  attackers  search  for  new  ways  to  circumvent  security 
controls  and  infiltrate  systems.  As  a  result,  security  practices  and  technologies  must  change 
accordingly  to  protect  against  new  vectors  of  attack.  To  apply  these  new  security  practices  and 
technologies  successfully,  cybersecurity  professionals  need  to  obtain  the  appropriate  knowledge, 
skills,  and,  eventually,  experience. 

An  organization  must  consider  several  factors  when  choosing  a  workforce  development  training 
program: 

1 .  The  training  program  needs  to  provide  the  workforce  with  the  knowledge,  skills,  and 
experience  that  are  most  relevant  to  their  job  duties. 

2.  The  training  program  needs  to  cultivate  a  high  level  of  proficiency  through  maximum 
development  of  knowledge,  skills,  and  experience. 

3.  Training  that  consumes  large  portions  of  an  individual’s  time  interferes  with  their  job  duties 
and  leads  to  lost  productivity. 

4.  The  scalability  of  a  training  solution  and  budget  limitations  restrict  the  amount  of  training  an 
organization  can  offer  to  its  workforce.  The  more  cost-effective  and  scalable  a  training 
solution  is,  the  larger  the  audience  it  can  reach. 
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Abstract 


For  most  established  organizations,  developing  and  maintaining  a  competent  cybersecurity 
workforce  needs  little  justification  and  is,  in  fact,  a  central  requirement  for  ensuring  resilient 
operations.  As  a  result,  these  organizations  invest  significant  resources  in  attempts  to  fulfill  this 
requirement.  However,  most  organizations  find  that  the  rapid  changes  and  dynamic  nature  of 
cybersecurity  make  keeping  their  workforce  up  to  date  a  very  challenging  problem.  This  report 
describes  a  traditional  model  commonly  used  for  addressing  this  challenge,  explains  some 
operational  limitations  associated  with  that  model,  and  presents  a  new,  continuous  approach  to 
cybersecurity  workforce  development. 
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1  Shortcomings  with  the  Traditional  Classroom  Training 
Model 


The  most  common  workforce  development  training  solution  is  the  traditional  classroom  training 
model.  While  this  training  model  is  easy  to  implement  and  is  widely  used,  there  are  a  number  of 
reasons  why  it  is  not  adequate  for  providing  effective,  large-scale  training  to  a  technical 
workforce. 

1 .  Traditional  classroom  training  is  not  ideal  for  developing  skills  and  experience  at  a  high 
level  of  proficiency.  Rather,  skills  and  experience  are  best  developed  in  environments  that 
closely  mirror  the  real-world  environments  where  they  will  be  applied.  For  a  cybersecurity 
workforce,  these  environments  include  elements  such  as  networks,  software  toolsets,  and 
user-generated  traffic. 

2.  Traditional  classroom  training  is  time  consuming.  Professional  training  seminars  and  courses 
often  consist  of  day-long  sessions  that  can  span  multiple  days.  Attendees  are  unable  to 
perform  their  job  duties  during  these  large  training  blocks,  so  organizations  lose 
productivity. 

3.  Traditional  classroom  training  does  not  scale  well,  nor  is  it  cost  effective  for  large 
organizations  that  have  employees  who  are  physically  distributed  across  different 
geographical  areas.  Specifically,  classroom  size — physical  dimensions  and  student-to- 
teacher  ratio — and  travel  costs  are  both  limiting  factors  that  restrict  the  amount  of  training  an 
organization  can  provide  to  its  workforce. 

4.  Traditional  classroom  training  is  not  optimal  for  rapidly  changing  fields  such  as 
cybersecurity,  where  practitioners  must  stay  abreast  of  the  most  current  trends,  technologies, 
and  practices  to  successfully  perform  their  job  duties.  Quickly  disseminating  new  and 
updated  training  courses  is  a  challenge  because  of  the  additional  time  and  costs  associated 
with  printing  new  material  and  having  instructors  learn  it. 

5.  The  limitations  of  the  traditional  classroom  training  model  (consumption  of  time  and  lack  of 
scalability)  translate  into  infrequent  training  opportunities  for  cybersecurity  workforce 
professionals.  As  a  result,  the  retention  and  mastery  of  knowledge  is  inhibited,  and  exposure 
to  the  most  current  cybersecurity  trends,  technologies,  and  practices  is  limited. 
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2  A  New  Approach  for  Developing  the  Cybersecurity 
Workforce 


The  CERT®1  approach  to  cybersecurity  workforce  development  builds  knowledge,  skills,  and 


experience  in  a  continuous  cycle  of  professional  development  (see  Figure  1).  Each  phase  focuses 
on  building  a  specific  area  of  development  that  is  leveraged  and  supplemented  by  the  next  phase 
of  development — with  the  purpose  of  reaching  some  end  goal.  The  end  goal  of  this  approach  is 
for  cybersecurity  professionals  to  use  relevant  knowledge,  skills,  and  experience  to  successfully 
and  effectively  perform  their  job  duties. 


Figure  1:  The  CERT  Approach  to  Cybersecurity  Workforce  Development 

There  are  three  main  development  phases  of  the  approach — knowledge  building,  skill  building, 

and  experience  building — and  an  evaluation  phase  that  allows  for  assessment. 

•  Knowledge  building  provides  a  solid  foundation  of  knowledge;  this  is  where  the 
fundamentals  and  concepts  of  a  particular  topic  area  are  learned. 

•  Skill  building  focuses  on  learning  how  to  apply  hands-on,  technical  skills  that  are  based  on 
the  foundational  knowledge  that  was  learned  in  the  previous  phase. 

•  Experience  building  develops  the  ability  to  adapt  and  successfully  apply  skills  in  changing 
and  unfamiliar  environments;  individuals  apply  knowledge  and  skills  in  real-world  scenarios. 

•  Evaluation  uses  performance  metrics  to  assess  individuals’  absorption  of  the  training 
material  and  identify  areas  of  improvement  for  continued  professional  development. 


CERT®  is  a  registered  mark  owned  by  Carnegie  Mellon  University. 
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2.1  Knowledge  Building 


The  goal  of  the  knowledge-building  phase  is  to  provide  individuals  with  a  foundation  of 
knowledge  that  will  facilitate  the  development  of  skills  and,  subsequently,  the  successful 
application  of  those  skills.  To  consistently  and  successfully  apply  a  skill,  an  individual  needs  to 
understand  the  basic  fundamentals  and  concepts  that  are  behind  it.  For  example,  when  people  are 
learning  how  to  drive  a  car  (the  skill),  they  first  learn  how  a  car  operates,  safety  practices, 
potential  hazards,  and  traffic  rules.  All  of  this  knowledge  is  learned  before  an  individual  gets  into 
the  driver’s  seat.  If  someone  attempts  to  drive  a  car  without  learning  this  foundational  knowledge, 
their  chances  of  being  able  to  successfully  operate  a  vehicle  are  low.  Similarly,  this  concept  holds 
true  for  developing  and  applying  cybersecurity  skills.  If  someone  does  not  understand  the  basic 
fundamentals  and  concepts  of  networking,  they  will  not  be  able  to  effectively  perform  skill- 
related  activities  such  as  reviewing  intrusion  detection  system  alerts,  monitoring  network  resource 
availability,  and  performing  basic  packet  capture  analysis.  As  a  result,  knowledge  building  is  a 
critical  first  step  in  cybersecurity  workforce  development  because  the  ability  to  effectively  apply 
cybersecurity  skills  directly  affects  job  performance  and  the  overall  security  of  an  organization. 

Knowledge  building  can  be  performed  in  several  different  ways.  Classroom  training  is  the  most 
traditional  method  for  knowledge  building  and  has  been  the  most  popular  form  of  professional 
development  in  many  fields,  including  cybersecurity.  Another  option  for  building  knowledge  is  to 
leverage  online  learning  solutions.  These  solutions  may  be  better  suited  for  professional 
development  for  these  reasons: 

1.  Online  training  is  extremely  cost  effective  because  (1)  it  is  usually  cheaper  since  expenses 
are  lower  for  the  online  training  providers,  and  (2)  organizations  do  not  have  to  pay  travel 
expenses  for  off-site  training. 

2.  Online  training  can  be  broken  into  manageable  segments  to  accommodate  work  schedules 
and  responsibilities.  Because  online  training  is  performed  asynchronously — meaning  that 
participation  only  depends  on  one  party  (the  participant) — it  is  possible  to  train  while  on  the 
job,  shifting  between  the  training  and  job  duties  without  them  interfering  with  each  other. 

3.  Online  training  enables  individuals  to  train  at  a  pace  that  corresponds  with  their  ability  to 
absorb  knowledge.  Some  individuals  may  prefer  a  full  day  of  training,  while  others  may  find 
it  more  beneficial  to  segment  training  into  two-  to  three-hour  blocks  over  several  days. 
Furthermore,  research  has  shown  that  online  learning  is  an  effective  and  viable  option  for 
professionals  and,  in  some  cases,  yields  better  results  than  traditional  face-to-face 
instruction.2 

2.2  Skill  Building 

The  purpose  of  the  skill-building  phase  is  to  develop  hands-on,  technical  skills,  based  on  the 
foundational  knowledge  learned  in  the  previous  phase,  which  will  be  used  to  effectively  perform 
job  duties.  It  is  in  this  phase  that  individuals  begin  to  apply  the  knowledge  they  have  been 
learning.  To  extend  the  analogy  about  learning  to  drive,  the  skill-building  phase  is  where  the 
person  gets  into  the  car  for  the  first  time  and  starts  to  learn  how  to  drive  it.  Because  driving  a  car 


U.S.  Department  of  Education,  Office  of  Planning,  Evaluation,  and  Policy  Development.  Evaluation  of  Evidence- 
Based  Practices  in  Online  Learning:  A  Meta-Analysis  and  Review  of  Online  Learning  Studies.  Washington,  DC, 
2009. 
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is  a  combination  of  various  skills,  the  driving  instructor  begins  by  developing  one  skill  at  a  time 
through  short,  simple  exercises.  This  approach  facilitates  greater  skill  proficiency  by  enabling  the 
student  to  concentrate  on  a  single  task.  For  example,  the  first  skill  someone  may  learn  is  how  to 
operate  the  gas  and  brake  pedals  by  slowly  accelerating  and  then  applying  the  brake.  After  the 
individual  has  mastered  that  skill,  they  progress  to  learning  how  to  make  turns  by  driving  in  a 
straight  line,  applying  the  brake  to  slow  down,  and  then  turning  around  a  cone — all  of  which  is 
done  in  an  empty  parking  lot.  Eventually,  the  student  will  have  developed  enough  skills  to 
combine  them  and  begin  driving  around  the  empty  parking  lot  similar  to  how  someone  would 
drive  on  the  road.  Developing  cybersecurity  skills  uses  a  similar  approach  regardless  of  what  the 
skill  is  related  to,  be  it  malware  analysis,  penetration  testing,  or  incident  response. 

Skill  building  is  an  integral  component  of  professional  development  for  the  cybersecurity 
workforce  because  sometimes  it  is  possible  for  individuals  to  perform  skill-related  activities  (i.e., 
activities  where  a  skill  needs  to  be  applied)  by  using  automated  software  tools.  Usually,  a  person 
needs  to  possess  a  skill — an  ability  typically  developed  through  training  and  experience — to 
perform  a  skill-related  activity.  For  example,  to  transport  a  piano  (the  skill-related  activity),  a 
person  needs  to  be  able  to  drive  a  truck  (the  skill).  However,  automated  software  tools  make  it 
possible  to  perform  skill-related  activities  in  technical  fields  without  possessing  the  skill  or  the 
foundational  knowledge  behind  it.  While  the  purpose  of  these  tools  is  to  improve  efficiency  by 
automating  manuals  tasks,  relying  on  these  tools  can  cause  an  erosion  of  skill  within  the 
cybersecurity  workforce  and  create  an  unskilled  population  of  cybersecurity  professionals.  For 
example,  there  are  numerous  digital  forensic  tools,  such  as  Autopsy,  EnCase,  and  Forensic  Tookit 
(FTK),  that  automate  certain  skill-related  activities,  such  as  carving  files  from  hard-drive  images, 
identifying  and  recovering  deleted  files,  and  parsing  file  system  information.  These  tools  are 
valuable  because  manually  performing  those  tasks  is  time  consuming  and  an  inefficient  use  of  a 
forensic  analyst’s  time.  However,  if  the  conditions  in  which  these  tools  normally  operate 
significantly  changes,  then  they  are  rendered  ineffective.  In  other  words,  if  a  file  system  is 
corrupted  to  a  certain  point,  these  tools  may  not  be  able  automate  activities  such  as  carving 
specific  files  from  a  hard  drive  image.  In  these  instances,  forensic  analysts  need  to  utilize  and 
adapt  their  skills  to  perform  these  tasks  manually.  An  unskilled  professional  who  relies  on  these 
tools  will  not  be  able  to  adapt  to  this  situation. 

Similar  to  the  driving  analogy,  cybersecurity  skills  are  best  developed  through  short,  narrowly 
focused  exercises  that  are  designed  to  transform  knowledge  into  the  ability  to  apply  it.  As  a  result, 
exercises  should  take  place  in  controlled  environments  (such  as  the  empty  parking  lot)  so  that 
individuals  can  focus  on  performing  specific  skill-building  activities  without  being  overwhelmed 
by  complex  information  technology  (IT)  infrastructures,  unpredictable  variables,  and  external 
stimuli — all  of  which  would  detract  from  skill  development  and  change  the  learning  environment. 
For  example,  suppose  an  individual  is  learning  how  to  use  Wireshark  to  capture  and  subsequently 
analyze  network  traffic.  In  the  skill-building  phase,  the  exercise  environment  may  involve  just 
two  systems — one  to  generate  some  simple  network  traffic  patterns  and  another  to  capture  them — 
with  the  skill-building  exercise  focusing  on  a  few  simple  activities,  such  as  performing  a  basic 
packet  capture,  applying  a  packet  capture  filter,  and  then  identifying  a  transmission  control 
protocol  (TCP)  handshake  in  the  capture  results.  As  an  individual  becomes  more  proficient  with 
these  skills,  the  difficulty  and  complexity  of  these  exercises  can  be  increased.  However,  it  is 
important  to  remember  that  the  exercises  still  occur  in  a  controlled  environment.  Once  an 
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individual  has  reached  a  certain  level  of  proficiency,  the  next  step  is  to  refine  those  skills  by 
applying  them  in  real-world  scenarios — otherwise  known  as  experience  building. 

2.3  Experience  Building 

The  goal  of  the  experience-building  phase  is  to  maximize  effective  job  performance  by  exposing 
individuals  to  real-world  scenarios,  events,  and  activities  that  are  similar  to  ones  they  will 
encounter  in  their  jobs.  In  the  previous  phases,  knowledge  and  skills  were  developed  in 
controlled,  focused  environments.  However,  actual  scenarios  often  occur  in  uncontrolled 
environments  that  are  complex,  unpredictable,  and  include  external  variables — all  of  which  can 
significantly  change  the  situation  and  operating  environment.  As  a  result,  experience  building  is 
needed  to  refine  knowledge  and  skill  so  that  it  can  be  successfully  applied  on  the  job  in  a  real- 
world  environment. 

Experience  is  defined  as  active  participation  in  events  or  activities  that  lead  to  the  accumulation  of 
knowledge  or  skill.  The  more  activities  or  events  an  individual  participates  in,  the  more 
experience,  and  therefore  more  knowledge  and  skill,  the  individual  gains.  Experience  is  valuable 
because  the  knowledge  and  skill  acquired  by  participating  in  specific  events  and  activities  can  be 
applied  when  similar  situations  are  encountered  again.  In  terms  of  the  driving  analogy,  the 
experience-building  phase  is  when  the  student  starts  to  learn  how  to  drive  in  traffic.  Until  now,  all 
knowledge  and  skill  building  has  occurred  in  controlled  environments.  Once  the  individual  has 
demonstrated  a  certain  level  of  proficiency  driving  in  an  empty  parking  lot  (skill  building),  the 
next  logical  step  in  development  is  to  expose  the  student  to  live  traffic  (experience  building). 
Although  it  is  possible  to  teach  advanced  driving  skills — such  as  parallel  parking,  making  three- 
point  turns,  and  navigating  road  hazards — in  an  empty  parking  lot,  there  is  complexity  and 
unpredictability  that  can  only  be  experienced  in  live  traffic  situations.  As  a  result,  if  someone 
attempts  to  drive  in  live  traffic  without  first  being  exposed  to  it  with  the  benefit  of  formal 
instruction,  they  will  have  difficulty  adapting  to  the  real-world  environment  and  are  more  likely  to 
have  an  accident.  This  concept  holds  true  for  refining  knowledge  and  skill  in  the  field  of 
cybersecurity.  Capturing  and  analyzing  network  traffic  in  a  controlled  environment  is  one  thing; 
applying  those  skills  in  a  real-world  environment  is  completely  different.  A  real-world  scenario 
could  include  multiple  networks,  large  quantities  of  traffic,  and  a  wide  variety  of  protocols. 
Additionally,  it  may  also  include  external  factors  that  cannot  be  controlled  or  predicted,  such  as 
attack  sophistication,  the  accuracy  of  third-party  information,  and  organizational  priorities. 

Being  exposed  to  the  unpredictability  of  real-world  scenarios  and  environments  in  the  experience¬ 
building  phase  also  develops  the  capability  to  successfully  adapt  and  apply  knowledge  and  skill  in 
changing  and  unfamiliar  situations.  Comprehension  of  knowledge  and  proficiency  with  skills 
increases  as  individuals  are  exposed  to  a  greater  variety  of  situations  in  which  they  must  apply 
them.  This  approach  is  widely  practiced  in  the  United  States  military,  where  comprehension  of 
knowledge  and  proficiency  with  skills  can  be  the  difference  between  life  and  death.  In  the 
military,  service  members  often  participate  in  exercises  that  simulate  real-world  scenarios  in 
which  unpredictable  and  changing  situations  are  introduced  to  force  the  participants  to  develop 
and  use  the  ability  to  adapt  and  apply  their  knowledge  to  the  given  situation. 

Although  on-the-job  training  is  a  common  method  used  for  experience  building,  it  is  not  always 
the  most  pragmatic  environment  for  learning.  First,  since  on-the-job  training  occurs  in  an 
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operational  work  environment,  efficient  and  effective  job  performance  usually  takes  priority  over 
training  and  development  needs.  Second,  some  situations  happen  so  infrequently  and  are  so 
important  that  they  are  not  practical  for  on-the-job  training,  such  as  situations  where  disaster 
recovery  and  business  continuity  operations  need  to  be  activated  (i.e.,  natural  disasters  or 
catastrophic  failures  of  infrastructure).  As  a  result,  an  alternative  method  for  experience  building 
is  to  simulate  real-world  environments  and  scenarios  that  cybersecurity  professionals  will 
encounter  on  the  job.  Until  recently,  creating  training  environments  that  mimicked  real-world  IT 
infrastructures  was  not  a  viable  solution  because  doing  so  was  a  costly  and  resource-intensive 
venture  that  involved  procuring,  configuring,  maintaining,  and  storing  large  amounts  of 
equipment.  However,  the  advent  of  virtualization  technologies  has  made  simulated  cybersecurity 
training  environments  a  cost-effective  option  for  experience  building.  For  example,  as  a  result  of 
virtualization,  an  entire  simulated  IT  infrastructure,  which  simulates  normal  network  traffic 
patterns  and  executes  real  attacks,  can  now  exist  on  a  single  server.  Additionally,  virtualized 
cybersecurity  training  environments  provide  a  much  more  scalable  and  accessible  solution  for 
experience  building  because  they  can  easily  be  replicated  and  made  available  over  the  internet. 

2.4  Evaluation 

The  goal  of  the  evaluation  phase  is  to  enable  a  continuous  cycle  of  professional  development  by 
assessing  knowledge  comprehension  and  skill  proficiency  and  by  making  it  possible  for 
organizations  to  accurately  catalog  its  employees’  knowledge  and  skills.  Continuous  professional 
development  is  particularly  important  for  the  cybersecurity  workforce  because  the  field  is 
constantly  changing — technologies  rapidly  evolve,  and  attackers  quickly  adapt  to  circumvent  the 
latest  security  practices.  This  constant  change  makes  a  formal  evaluation  mechanism  critical  for 
cybersecurity  workforce  development  methodology  because  organizations  need  to  be  able  to 
systematically  ensure  that  their  staff  maintains  the  knowledge,  skills,  and  experience  needed  to 
adapt  to  these  changes. 

Once  an  individual  has  gone  through  the  three  development  phases  of  the  approach — knowledge 
building,  skill  building,  and  experience  building — it  is  important  to  assess  the  level  of  knowledge 
comprehension  and  skill  proficiency  that  was  achieved  through  the  training.  This  assessment 
should  be  linked  to  instructional  objectives,  which  define  what  an  individual  should  know 
(knowledge)  or  be  able  to  do  (skill)  after  they  go  through  a  particular  training  course  or  module. 
As  a  result,  instructional  objectives  serve  as  excellent  metrics  for  assessing  knowledge  and  skill 
(see  Table  1).  For  example,  one  instructional  objective  for  an  incident  response  training  course 
could  be  that,  given  access  to  network  monitoring  devices  that  capture  both  normal  and  abnormal 
network  traffic,  the  student  will  be  able  to  identify  web  server  vulnerability  scanning  activities  in 
at  least  intrusion  detection  system  (IDS)  and  web  server  logs.  In  this  example,  the  instructional 
objective  clearly  defines  the  skill  being  developed  (i.e.,  identifying  web  server  vulnerability 
scanning  activities)  and,  as  a  result,  skill  proficiency  can  be  measured. 
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Table  1:  Four  Components  of  an  Instructional  Objective 


Component 

Explanation 

Audience 

Who  is  the  training/exercise  aimed  at? 

Behavior 

What  do  you  expect  the  audience  to  be  able  to  do?  This  is  an  overt,  observable  behavior,  even 
if  the  actual  behavior  is  mental  in  nature  (i.e.,  comprehension  of  knowledge). 

Condition 

Under  what  circumstances  will  the  learning  occur? 

Degree 

What  criteria  need  to  be  met  for  this  objective  to  be  achieved?  For  example,  is  the  objective 
achieved  with  total  mastery  (100%)  or  when  a  minimum  standard  (70%)  is  met? 

Assessing  knowledge  and  skill  during  the  evaluation  phase  determines  whether  an  individual  has 
achieved  the  desired  level  of  knowledge  comprehension  and  skill  proficiency  from  the  training. 
The  assessment  will  be  used  to  determine  the  next  training  cycle  for  an  individual.  If  an  individual 
has  achieved  the  desired  levels  of  knowledge  and  skill,  they  are  ready  to  move  to  more  advanced 
training  or  a  different  subject  matter.  However,  if  an  individual  does  not  achieve  the  desired  levels 
of  knowledge  and  skill,  the  assessment  can  be  used  to  identify  areas  of  weakness  that  need  to  be 
improved.  In  either  situation,  the  goal  is  to  provide  individuals  with  a  path  for  continued 
professional  development. 

The  evaluation  phase  also  provides  a  mechanism  that  organizations  can  use  to  obtain  a  better 
understanding  of  the  knowledge  and  skills  that  its  workforce  possesses,  which  benefits  both  the 
organizations  and  its  workforce.  First,  organizations  will  be  able  to  provide  more  meaningful  and 
relevant  training  to  its  workforce  because  the  information  obtained  from  the  evaluation  phase  will 
make  it  easier  for  organizations  to  identify  training  needs.  As  a  result,  cybersecurity  professionals 
within  the  organization  will  have  more  training  available  that  furthers  professional  development. 
Second,  organizations  will  be  able  to  better  maximize  job  performance  by  ensuring  that 
individuals  possess  the  right  knowledge  and  skills  needed  to  effectively  perform  their  job  duties. 
Deficiencies  in  knowledge  and  skills  will  be  easier  to  identify,  and  organizations  can  address  any 
shortcomings.  Third,  organizations  can  use  the  information  about  its  employees’  knowledge  and 
skill  to  help  mitigate  operational  risk.  Specifically,  this  information  can  be  used  to  identify  and 
avoid  single  points  of  failure  with  respect  to  critical  knowledge  and  skills  within  the  organization. 
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3  Case  Study 


The  following  case  study  is  an  example  of  how  the  approach  can  be  implemented.  The  case  study 
not  only  highlights  a  real-world  implementation  of  the  cybersecurity  workforce  development 
approach  but  also  demonstrates  that  the  approach  is  flexible  and  can  be  used  to  meet 
organizational  training  needs.  The  case  study  leveraged  the  Carnegie  Mellon®  Software 
Engineering  Institute’s  Virtual  Training  Environment3  (VTE)  and  the  CERT  Exercise  Network4 
(XNET)  technologies  (Table  2)  to  implement  the  different  phases  of  the  approach. 


Table  2:  CERT-Developed  Training  Technologies 


Technology 


Description 


VTE  Combines  the  components  of  traditional  classroom  training  with  the  benefits  of  web-based 

training.  Users  can  conveniently  access  VTE  from  their  own  computers  and  participate  in 
training  courses  at  their  own  pace.  VTE  contains  a  library  of  instruction  and  reference 
material  about  information  assurance,  incident  response,  computer  forensics,  and  other 
vital  cybersecurity  topics.  Instruction  includes  lectures,  slides,  and  written  material  for 
knowledge  building  and  hands-on  exercises,  lab  books,  and  technical  demonstrations  for 
skill  building. 


XNET 


Designed  to  address  the  challenges  of  realism  and  scalability  of  scenario-based 
cybersecurity  exercises  and  simulations.  XNET  provides  a  platform  for  experience  building 
by  enabling  instructors/trainers  to  create  customized,  full-scale  cybersecurity  exercises  that 
simulate  real-world  scenarios  and  environments.  Multiple  instantiations  of  the  same 
exercise  can  be  deployed  simultaneously  to  accommodate  a  large  number  of  participants, 
and  the  XNET  participant  console  is  accessible  via  the  internet  to  maximize  the 
accessibility  of  exercises. 


3.1  The  U.S.  Air  Force:  Unit-Level  Force  Development 

In  2009,  CERT  partnered  with  a  U.S.  Air  Force  (USAF)  Cyber  Operations  Squadron  to  develop  a 
way  to  train  network  warfare  teams,  or  crews,  for  computer  network  defensive  operations  and  to 
measure  mission  readiness  afterwards.  One  goal  of  this  project  was  to  develop  the  incident 
responders’  knowledge,  skill,  and  experience  in  basic  digital  forensics  to  reduce  the  workload  of 
the  forensic  analysts  and  facilitate  more  effective  collaboration  between  the  incident  response 
teams  and  the  forensic  analysts. 

3.1.1  Segment  One:  Evaluation 

This  particular  implementation  of  the  approach  to  cybersecurity  workforce  development  started 
with  the  evaluation  phase,  which  was  used  to  determine  the  kind  of  knowledge,  skills,  and 
experience  that  was  to  be  developed  in  the  subsequent  segments  of  the  training.  During  this 
segment,  XNET  was  used  to  conduct  a  standard  squadron  technical  evaluation  of  the  real-time 
analysts  (incident  responders).  Each  analyst  was  provided  with  a  virtualized  representation  of 
their  workstation  and  network  environment,  which  included  the  same  tools  they  used  on  the  job 


3  http://vte.cert.org 

4  http://xnet.cert.org 

®  Carnegie  Mellon  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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and  a  sampling  of  the  Air  Force  Global  Information  Grid  (GIG)  infrastructure.  The  analysts  were 
presented  with  a  real-world  cybersecurity  scenario  and  were  observed  by  unit  evaluators,  who 
injected  attacks  and  other  inputs  into  the  scenario  during  the  session. 

After  the  individual  technical  evaluations,  real-time  analysts  were  paired  up  and  presented  with  a 
more  complex  team-based  incident  response  evaluation  in  XNET.  The  scenario  encompassed  the 
anatomy  of  a  real  attack — reconnaissance,  botnet  and  malware  staging,  data  exfiltration,  and 
massive  communications  disruption — and  involved  more  than  100  virtualized  computers  and 
infrastructure  devices.  Analysts  were  provided  with  a  virtual  representation  of  their  workstations 
and  a  variety  of  incident  detection,  response,  and  forensic  tools.  Similar  to  the  individual  technical 
evaluations,  evaluators  and  instructors  observed  the  teams’  performance  during  the  scenario. 

3.1.2  Segment  Two:  Knowledge  and  Skill  Building 

During  segment  two,  VTE  was  used  to  develop  the  knowledge  and  skills  that  would  be  needed  for 
the  final  capstone  exercise.  Knowledge  building  was  achieved  by  providing  the  participants  with 
lectures,  slides,  and  technical  demonstrations  about  various  incident  response  and  digital  forensic 
topics.  Skill  building  was  accomplished  by  completing  hands-on  technical  labs  that  were  narrowly 
focused  on  developing  specific  skills.  For  example,  all  analysts  were  required  to  complete  a  lab 
that  walked  them  through  the  step-by-step,  forensically  sound  process  of  capturing  a  hard  drive 
image  from  a  compromised  server  and  performing  initial  forensic  analysis  of  the  image.  VTE 
proved  to  be  an  ideal  environment  for  knowledge  and  skill  building  because  it  provided  (1)  a 
robust  technical  capability  for  delivering  online  training  content  and  (2)  a  controlled  hands-on 
learning  environment,  which  is  conducive  for  developing  knowledge  and  skill. 

3.1.3  Segment  Three:  Experience  Building 

The  final  segment  of  training  culminated  in  a  capstone  exercise  in  XNET  that  required  the 
participants  to  use  and  apply  their  knowledge  and  skill  in  a  real-world  scenario  and  modeled 
infrastructure  environment.  The  goal  of  this  capstone  event  was  to  further  refine  the  participants’ 
knowledge  and  skill  by  exposing  them  to  an  exercise  that  introduced  complexity  and 
unpredictability  into  the  equation.  The  capstone  exercise  built  on  the  team-based  technical 
evaluation  in  segment  one  and  required  the  five-person  teams,  composed  of  four  real-time 
analysts  and  one  forensic  analyst,  to  coordinate  and  conduct  initial  incident  response  and  forensic 
analysis  of  the  cyber-attacks  detected  in  segment  one.  The  teams  were  provided  with  a  suite  of 
tools  and  were  required  to  move  outside  of  their  regular  mission  scope — analyzing  real  time 
alerts — and  perform  digital  forensics  tasks,  such  as  event  inventory  and  correlation,  live  system 
data  acquisition,  and  log  file,  hard  drive,  and  memory  image  forensic  analysis.  XNET’s  automated 
evaluation  capabilities  were  used  to  conduct  staged  performance-based  assessments  as  well  as 
collect  situational  awareness  reporting. 
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4  Conclusion 


Traditional,  classroom-based  training  models  are  effective  for  certain  types  of  learning  but  are  not 
ideal  for  satisfying  the  developmental  needs  of  the  cybersecurity  workforce.  The  CERT  approach 
to  cybersecurity  workforce  development  is  divided  into  continuous  phases  of  development  that 
progressively  builds  an  individual’s  knowledge,  skills,  and  experience  in  ways  that  are  relevant  to 
their  job  duties.  The  approach  combines  the  concepts  of  classroom-based  models  with  the 
flexibility  of  online  platforms  and  adds  another  layer  of  development  that  is  particularly  beneficial 
for  professionals  in  the  field.  Specifically,  it  incorporates  a  focus  on  experiential  learning — 
providing  real-world  scenarios  that  enable  participants  to  apply  their  knowledge  and  skills  in 
situations  and  environments  they  may  face  on  the  job.  As  a  result,  the  CERT  approach  to 
cybersecurity  workforce  development  offers  organizations  a  comprehensive,  targeted,  cost- 
effective  training  option  that  can  be  tailored  to  their  needs. 
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